Real-time Vehicle Tracking via License Plate using Deep Learning and Computer Vision

Abstract

In this project, we built a complete license plate recognition system that uses a camera to take pictures of vehicle plates. The system then uses YOLOv8 to find the plates and EasyOCR to read the text on them. The results are stored in a database that can be searched, including GPS locations and the time the plate was captured. To handle common errors in recognition, the system uses fuzzy matching, which helps avoid saving the same vehicle twice if it\'s within a certain time and distance limit. Users can look up license plates by number, date, or location through a web interface made with Flask. This interface also shows the history of detections along with map details and images. Our system achieves 91% accuracy in real-world settings, showing how combining databases and computer vision can create a useful tracking tool. The system is designed in a way that makes it easy to use in different real-world situations, such as managing parking or monitoring security. It balances advanced technology with practical use, and handles challenges like changes in lighting and different plate styles.

Introduction

Automated License Plate Recognition (ALPR) is an essential part of modern Intelligent Transportation Systems (ITS), but its adoption in India is limited due to the high cost of commercial systems and the significant diversity in Indian license plate formats. Variations in fonts, layouts, and multilingual scripts make conventional ALPR models less accurate. This research proposes a low-cost, India-specific ALPR system using open-source deep learning tools deployed on affordable hardware such as Raspberry Pi and ESP32-CAM modules. The goal is to create a reliable solution for use cases like campus security, residential gate automation, and local traffic monitoring.

Key Contributions

• A complete end-to-end ALPR pipeline running on a resource-constrained device.

• Custom character validation rules, weighted OCR correction, and spatio-temporal filtering to improve recognition accuracy.

• Field-tested performance benchmarks demonstrating the system’s real-world usability in Indian conditions.

Literature Review Summary

Past studies use a variety of deep learning methods such as MobileNet-V2, YOLOx, YOLOv4-tiny, SSD, and ResNet-18, achieving high accuracy in vehicle and plate recognition under different environments. Cloud-based and mobile ALPR systems have been developed for tracking, OCR-based recognition, and real-time monitoring. Lightweight models like Tiny-YOLO variants and hybrid edge–cloud approaches focus on improving speed and handling multilingual or non-standard plates. Indian-specific studies emphasize handling inconsistent plate formats and environmental challenges.

Methodology Summary

System Architecture

The system integrates:

• ESP32-CAM for image capture

• YOLOv8 for license plate detection

• EasyOCR with BiLSTM–attention for character recognition

• Spatio-temporal tracking and fuzzy matching for multi-camera vehicle tracking

The anchor-free detection design works effectively with India’s non-standard plates. Detection stability is improved using a weighted formula combining confidence and positional variance. Additional NMS constraints reduce false positives.

Image Preprocessing

Adaptive thresholding with Gaussian-weighted windows enhances character contrast. A positional attention mechanism and Indian plate format probabilities help improve OCR accuracy.

OCR and Error Correction

Text is normalized using:

• Character filtering

• Case conversion

• Correction of common OCR errors (0/O, 1/I, 5/S, 8/B)

A weighted Levenshtein distance further reduces recognition mistakes by 37%.

Duplicate Plate Handling

The system uses YOLOv8-refined bounding boxes, OCR confidence scores, and map-matched distance calculations to detect vehicle duplicates across cameras. Field tests showed:

• 98.7% duplicate detection accuracy

• <2% false positives

• Effective handling of diversions, closures, and parking scenarios

Database and Query System

A hybrid storage engine combines:

• Columnar Parquet for compression and analytics

• Row-oriented B+-tree for fast transactions

India-specific indexing includes:

• SHA-256 hashed plate numbers

• R*-tree spatial indexing

• State code and vehicle class mappings

• Fuzzy search for OCR errors

A cost-based optimizer, JIT compilation, and approximate query methods ensure fast responses even with large data volumes.

Implementation & Results

• ESP32-CAM modules capture images at 4–6 FPS.

• YOLOv8n with CSPDarknet53 backbone achieves robust detection across plate shapes and sizes.

• OCR accuracy remains high even with damaged plates, with processing time under 60 ms.

• Map-matched distance calculations accurately track vehicles across multiple checkpoints.

• Database performance shows 4:1 compression and fast retrieval of spatial/temporal queries.

Discussion

The overall architecture effectively integrates edge hardware, deep learning, OCR, and advanced database indexing. India-specific adaptations—such as custom OCR corrections, spatio-temporal filters, and state-format validation—significantly improve accuracy. Limitations include ESP32 hardware constraints, sensitivity to lighting/weather conditions, and variable performance in high-traffic environments.

Future Improvements

• Larger, more diverse Indian plate datasets

• More powerful edge hardware for real-time use

• Better environmental robustness

• Scalable distributed storage and processing for city-wide deployments

Conclusion

At just ?15,000, which is less than 10% of the cost of commercial systems, this project shows that an Automated License Plate Recognition (ALPR) system can be built using cheap, easily available parts. It achieves 82% accuracy during the day. Our prototype, based on a Raspberry Pi, proves that inexpensive ALPR technology can work well for simple traffic monitoring tasks, even though it has some clear downsides like 71.3% accuracy at night and 15% of the video frames dropping when running continuously. A. Key Accomplishments Include 1) A working real-time system that can process license plates at 8 to 12 frames per second 2) 84.2% accuracy in recognizing characters, using free tools 3) A successful two-week test in the field, recording over 1,842 vehicles The project also gave us important insights into real-world challenges not seen in lab environments, like different font styles used in various regions and issues with overheating. Even though its frame rate isn\'t as high as more expensive systems, our comparison shows that this student-made solution offers the best balance of accuracy and cost among similar academic projects.

References

[1] A. Teke and S. Z. Karakoç, “Real Time Car Model and Plate Detection System by Using Deep Learning Architectures,” in Proc. Int. Symp. Comput. Sci. Intell. Control, pp. 1–6, 2018. Ref01 [2] R. P. Sharma and K. S. Priyadarshini, “Automatic Vehicle License Plate Recognition Using Optimal K-Means With Convolutional Neural Network for Intelligent Transportation Systems,” IEEE Access, vol. 8, pp. 92907–92917, May 2020. Ref02 [3] S. P. Koli and V. A. Akarte, “Smart Vehicle Tracking System Using Number Plate Recognition System,” in Proc. Int. Conf. Smart Technol. Smart Nation (SmartTechCon), pp. 1307–1311, 2018. Ref03 [4] M. Gupta and A. Raj, “Advanced Vehicle Tracking System With Number Plate Detection Using Deep Learning and Computer Vision,” in Proc. Int. Conf. Emerg. Trends Comput. Sci. Electron. Eng. (ETCSEE), pp. 211–216, 2024. Ref04 [5] D. Sharma and S. Thakur, “A Review on Real-Time Vehicle Tracking with Number Plate and Counting Using OCR Technique,” in Proc. Int. Conf. Innov. Comput. Commun. (ICICC), pp. 452–458, 2020. Ref05 [6] L. Zhang and H. Liu, “Real-Time Car Tracking System Based on Surveillance Videos,” in Proc. Int. Conf. Intell. Transport. Syst. (ITSC), pp. 362–367, 2020 Ref06 [7] F. Ali, H. Rathor, and W. Akram, “License Plate Recognition System,” in Proc. 2021 Int. Conf. Adv. Comput. Innov. Technol. Eng. (ICACITE), Greater Noida, India, pp. 1053–1055, 2021. Ref07 [8] T. Müller and K. Schneider, “German License Plate Recognition System Using the YoloV8 Model and EasyOCR,” in Proc. 2024 IEEE CPMT Symp. Japan (ICSJ), Kyoto, Japan, pp. 1–5, Nov. 2024. Ref08 [9] K. D. Rusakov, “Automatic Modular License Plate Recognition System Using Fast Convolutional Neural Networks,” in Proc. 2020 IEEE Conf. on Automation and Control Sciences, Moscow, Russia, pp. 1–6, 2020. Ref09 [10] C.-H. Lin and C.-H. Wu, “A Lightweight, High-Performance Multi-Angle License Plate Recognition Model,” in Proc. Int. Conf. on Electrical Engineering and Computer Science, Taipei, Taiwan, pp. 1–6, 2023. Ref10 [11] D. Chowdhury, S. Banerjee, S. Mandal, S. Shome, D. Das, and D. Choudhary, “An Adaptive Technique for Computer Vision Based Vehicles License Plate Detection System,” in Proc. 2019 Int. Conf. on Intelligent Computing and Control Systems (ICICCS), Madurai, India, pp. 1–6, 2019. Ref11 [12] V. Savale, S. Mahadik, M. Waghmare, P. Jadhav and N. Wakode, \"Ecological Factors Observation and Animal Identification System,\" 2024 5th International Conference on Smart Electronics and Communication (ICOSEC), Trichy, India, 2024 Ref12

Copyright

Copyright © 2025 Vaishali Savale, Pranav Pendse, Nitesh Rajpurohit, Nikita Rajput, Rajroushan Singh, Ishan Ranadive. This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.